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Abstract. The clustering properties of Jack polynomials are relevant in the theoret- 
ical study of the fractional Hall states. In this context, some factorization properties 
have been conjectured for the (g, i)-dcformcd problem involving Macdonald polynomi- 
als (which are also the quantum eigenfunctions of a familly of commuting difference op- 
erators with signifiance in the relativistic Ruijsenaars-Schneider model). The present 
paper is devoted to the proof of this formula. To this aim we use four families of 
Jack/Macdonald polynomials: symmetric homogeneous, nonsymmetric homogeneous, 
shifted symmetric and shifted nonsymmetric. 



1. Introduction 

The symmetric (homogeneous) Jack polynomials are relevant in the study of the 
quantum many-body wave functions. In particular, fractional quantum Hall states 
of particles in the lowest Landau levels are described by such polynomials [Sj HJ [5l 
[6]. Pioneered by Laughlin [23], the theoretical study of the fractional quantum Hall 
states use multi-variable polynomials [17] describing the full many-body state of the 
interacting electrons on a plane or on a sphere (In the case of the sphere, the polynomials 
appear after stereographic projection). The special polynomials that are relevant in this 
context are not general solutions of the true eigenvalue problem involving the Coulomb 
interaction but they are constructed to be adiabatically related to the true eigenstates. 
The most famous example is the Laughlin wave function which is the cube of the 
Vandermonde determinant ni<j(^« ~ ^j)^ the variables representing the particles. 
It is known to be a good approximation of the true state of electrons for the lowest 
Landau level with filling factor |. Another interesting and celebrated example is the 
Moore-Read Pfaffian ^ |28j 

*M.:=pf(^)n(-^--.) 

which is one of the candidates to approximate the system for the filling factor | and is 
the polynomial of smallest degree belonging to the kernel of the operator which forbids 
three particles to be in the same place 

i<j<k 
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This operator can be naturally generalized to operators which forbid k particles in 
the same place, the lowest degree polynomials in its kernel provide other examples of 
wave-functions called Read-Rezayi [221 EO] states: 

k 

l=\ l<ii<ji<N 

This family of wave functions is composed with multivariate symmetric polynomials 
with some additional vanishing conditions, namely wheel conditions. These polyno- 
mials, whose study was pioneered by Feigin, Jimbo, Miwa and Mukhin, are proved to 
belong to a family of Jack polynomials with negative rational parameter [121 ES] • Fol- 
lowing the notations of Bernevig and Haldane [3], HI [5], IH |7] , these polynomials depend 
upon a parameter a and a configuration of occupation numbers [tiq, Wi, . . . ]. To recover 
the standard notation of symmetric functions [25], the set of occupations defines a de- 
creasing partition A = [Ai > A2 • ■ ■ > A^v] of the number N of particles. The partition A 
corresponding to the vector [tiq, ^1, . . . ] is such that rij is the multiplicity of the number 
i in A. The relevant wave functions belong to the kernel of the differential operators 

^+'■=11-^. ^- •= ^0^^^^ - 

i i i 

where A''^ denotes the number of flux quanta. In the case of the sphere, through the 
stereographic projection, the natural action of the SU{2) rotations on the quantum 
states is translated in an action of L+, L_ and = \NN^ — ^i'^.- O^i^ of the au- 
thors (J.-G.L.) with Th. Jolicceur[18] described a family of Jack polynomials belonging 
in the kernel of L_|_. More precisely, they investigate a (g, t)-deformation of the problem 
involving Macdonald polynomials p5]. 

Bernevig and Haldane [4jj identified (/c, r)-clustering properties of some wave functions 
which relate the functions with Nk variables with those with N[k — 1) by a factorization 
formula of the kind: 

N-k 

^^^^''\xi, . . . , y,...,y)= JJ (x^ - yy¥''''\xi, . . . , X(^N-i)k)- 



xk 



=1 



In particular, this is the case for the Read-Rezayi states: 



N-k 



^Ruixi, . . . , x(iv-i)fc, y, . ..jy) = n ^ yf¥'''''\xi, . . . , x(N-i)k] 



xk ^=1 



Furthermore, Bernevig and Haldane [3] showed the connection between the Read-Rezayi 
states and staircase Jack polynomials -P^"'*: 

^_Ri?(^) = -^[{2(Af-l))*,...,2'=,0'=] 

The link between Jack polynomials and quantum Hall states was proven by B. Estienne 
et a/.[TOl|TT]. Recently, Baratta and Forrester [2] proved that staircase Jack polynomials 
for some negative rational parameter satisfy the clustering conditions. More specifically, 
they stated the result in terms of Macdonald polynomials and recovered the property 
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as a limit case. In the same paper, they conjectured very interesting identities for 
rectangular partitions (the initial case of staircase with only one step): 



N r-1 

Pr3{y,yq'',...,yq " ,XN-g, ■ ■ ■ ,Xn) = [[ [[{xe-q'-y), 

where a is a certain negative rational number. The aim of our paper is to prove this 
conjecture by using some other families of Macdonald polynomials. Indeed, we will use 
four families of Jack/Macdonald polynomials: symmetric homogeneous, nonsymmetric 
homogeneous, shifted symmetric, shifted nonsymmetric. The Macdonald polynomi- 
als are a two-parameter deformation of the Jack polynomials which can be used to 
finely understand equalities involving Jack polynomials. This is due to the fact that 
they appear in the representation theory of the double affine Hecke algebra (the Jack 
polynomials are a degenerated version which involves the algebra of the symmetric 
group). Furthermore they admit a physical interpretation as the eigenfunctions of the 
Macdonald- Ruijsenaars operator 



^ ^ Xi _ pXj+rj 



i=l j=i 



which is a relativistic version of quantum Calogero-Moser system (see [HI [32]). The 
parameters q and t are related to the parameters h and i] hj q = e^/^ and t = e''^^. 

One approach to developing a theory of orthogonal polynomials of several variables 
of classical type led to symmetric and nonsymmetric Jack polynomials. The associated 
mechanism involves the symmetric group and Young tableaux. Just as the theory of 
hypergeometric series is extended to basic hypergeometric series, the theory of Jack 
polynomials is extended by the Macdonald polynomials. Here the symmetric group is 
replaced by its Hecke algebra. The orthogonality of the polynomials comes from their 
realization as eigenfunctions of a certain set of commuting operators. Generally these 
polynomials have coefficients which are rational functions of the parameters {q,t), as- 
sumed not to be roots of unity. Later a further generalization was developed, namely 
the theory of shifted Macdonald polynomials, which are nonhomogeneous polynomi- 
als defined by the property of vanishing on certain points, corresponding to so-called 
spectral vectors. This vanishing property leads to expressions of the polynomials as 
products of linear factors in various special cases. 

It turns out that for special values of the form q"'t'' = 1 for positive integers a,b, a 
shifted Macdonald polynomial collapses to its highest degree term, in which case, it 
agrees with an ordinary homogeneous Macdonald polynomial. The label of the polyno- 
mial has to satisfy certain restrictions for this to be possible. These parameter values 
result in the polynomials being of "singular" type. In this paper we concentrate on the 
rectangular polynomials, meaning those whose leading term is of the form {xiX2 ■ ■ ■ Xk)^ ■ 
Our results provide factorizations which are interpreted as clustering properties. 

The paper itself starts with an overview of the four different types of Macdonald poly- 
nomials (shifted, ordinary, symmetric, nonsymmetric) and the algebraic structure of the 
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associated operations. The technical machinery comprises tableaux, adjacent transpo- 
sitions, raising operators, and the Yang-Baxter graph. Then there is a presentation 
of the binomial formulae of Knop and Sahi which show how to expand nonhomoge- 
neous Macdonald polynomials in terms of homogeneous ones. These series are then 
specialized to the rectangular versions and this leads to the proofs of our main results. 
These all concern the case of variables with k < N/ 2 and the parameters satisfying 
qa^N-k+i _ ^ (-^ith a = m — 1 for the symmetric type and a = m for the nonsymmetric 
type) but no such relation with smaller exponents. In closing there is a discussion of 
future research in the direction of polynomials of staircase type; this phrase refers to 
the pictorial representation of the label of the polynomial, for example (4, 4, 2, 2, 0, 0). 

2. Macdonald polynomials 

Macdonald polynomials arc special functions which are involved in the representation 
theory of the affinc Hcckc algebra. This paper is devoted to the case of the symmet- 
ric group (type A^^i). Up to normalization Macdonald polynomials are defined to 
be the simultaneous eigenfunctions of some operators which are a (g, i)-deformation 
of the Cherednik operators. We study four variants of these polynomials: symmetric, 
nonsymmetric, shifted symmetric and shifted nonsymmetric. Symmetric Macdonald 
polynomials are indexed by decreasing partitions A whilst the nonsymmetric ones are 
indexed by vectors v G N^. In the aim to simplify the expression arising in the compu- 
tation, we use the notion of legs and arms of a cell in the Ferrers diagram of a vector. 
These numbers are (classically) defined by 

Q_^^(i,j) := v\i] -j 

and 

-^vihj) #{k<i-j< v[k] + 1 < v\i]} + #{i<k:j< v[k] < v\i]}. 
Note if = A is a partition then Jl is the classical leg-length: 

where A' denotes the conjugate of A. 

The leg-length and arm-length are used to define the {q, i)-hook product of v with 
argument z: 

' J ^ 



hqA^^^) = n 

{i,j)ev 



I — Zq ' = v(i,j)-j--^ vii,j) 

V J 



We need also the arm-colength and the leg-colength 

^.(j)=J-l 

and 



l^(i) ^{k<i: v\i] < v[k]} + i^{i<k: v\i] < v[k]}. 
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Note the rank function of v: 

r, = [1+1,(1),. ..,1 + 1, (iV)] 

is a permutation of &n- 

2.1. AfRne Hecke Algebra. We use the notation of Lascoux, Rains and Warnaar[22J. 
Let us recall it here. Let > 2 be an integer and X = {xi, . . . ,xn} be an alphabet 
of formal variables. We consider the operators Tj acting on Laurent polynomials in the 
variables Xj by 

(1) T, = t + {s,-lp±^, 

where Si is the elementary transposition permuting the variables Xi and Xj+i. These 
operators act on the right and in particular we have 

(2) ITi = t and Xj+iTj = Xi. 

More precisely, Tj is the unique operator that commutes with symmetric functions in 
Xi and Xj+i satisfying ([2]). The operators Tj satisfy the relations of the Hecke algebra 
of the symmetric group: 

i-'i+l-'i = -'i+l-'i-'j+l 

(3) TiTj = TjTi for |« — j| > 1 

(r,-t)(T, + i) = o 

Together with multiplication by the variables Xi and the affine operator r defined by 
f{x)T = /(xAf/g, Xi, . . . , xtv-i), they generate the affine Hecke algebra of the symmetric 
group. More precisely, 

(4) nN{q, t) = C(g, t) [xf , . . . , x± , T±, . . . , T±_„ r]. 

2.2. Nonsymmetric Macdonald polynomials. The nonsymmetric Macdonald poly- 
nomials {Ey)y(z^N are defined as the unique basis of simultaneous eigenf unctions of the 
(g, t)-version of the Cherednik operators defined by 

(5) ir.= t^-%-i...T,TT^\...Tr\ 

such that E^ = x'' + with x"" = x"^^^ . . . x^t^l ii v = [v[l], . . . , v[N]] and ^ 

denotes the dominance order on vectors which is based on the dominance order for 
partitions: 

A /i if and only if Vi, Ai + ■ • • + Aj < /ii + ■ • ■ + /Xj. 

This order is naturally extended to vectors with the same definition. The dominance 
order ^ for vectors is 

u ^ V if and only if either :<£, or {u^ = and u :<o v) 

where denotes the unique nonincreasing partition which is a permutation of u. 
Note Cherednik operators commute with each other and generate a maximal commu- 
tative subalgebra of T-Lj^^q^t). 

The corresponding spectral vectors are given by Spec,[i] = -j^^ with 
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We recall also the (g, t)-Dunkl operators 

Dn = {1- ^n)x-^\ a = tT-^D,+iTr\ 

Set c^g,t) = hg^t{v,q). We will use another normalization which is useful when sym- 
metrizing: 

where n{v) = ^(.^^.^^^ J„(i, j). 

KnopjlSj defined and studied the polynomials 

(7) £, = c:{q,t)E, 

with c'^{q, t) = hq^t{v, qt). The expansion of £y on the monomial basis is known to have 
integral coefficients in Z [g, t, [T5]. 

2.3. Nonsymmetric shifted Macdonald polynomials. The definition of nonsym- 
metric shifted Macdonald polynomials M„ is quite similar to those of the Macdonald 
polynomials E^: This is the unique basis of simultaneous eigenfunctions of the Knop- 
Cherednik operators defined by 

(8) := ti-*T,_i . . . rir(l - —)T^\ . . . Tr^ + -. 

Xm Xi 

such that M„ = g-"'(^)a;^ + ^^^^ a^^^x" with x'" = x^l^l . . . x^'l^l if v = [v[l], . . . ,v[N]], 



E 

(l,j)GD 



n'{v) = > ^ D — 



and ^ denotes again the dominance order on vectors. 

Note that, initially, the dominance order is defined only for vectors with the same norm. 
We can straightforwardly extend it for any vectors by adding the condition u -< v 
when \u\ < \v\. We remark that the Sahi binomial formula [33j, more precisely one of 
its consequences [22] (Corollary 4.3), together with a theorem of Knop [16] (see also 
equation ( 126|) ) imply that if |m| < \v\ then x^ has a non null coefficient in the expansion 
of only if C f It follows that the natural extension of the dominance order for 
any pairs of vectors, i.e.: 

u ^ V if and only if either :<£> or (m^ = and u :<d v), 

matches with the order appearing in the expansion of M^. The reader can refer to 
appendix |A] for a discussion about the notations. 

Note operators C,i and Sj can be constructed in a similar way, as shown in the following 
proposition. 



Proposition 1. Fori <i < N, = tTr^ii+iTr^ and Ei = tTr^E,+iT- 



-1 
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Proof. The first part is obvious. For the second part we have 
for any polynomial p, and 

V Xi^i / IXi 

thus t7^"^Sj+iTj~^ = Sj. 



Also we have: 
Proposition 2. Ei = + Di for 1 < i < N . 
Proof. Start with i = N. For any polynomial p 

pE^ =t->T^-i...T,r(l-^)+^ 

= P^N + ip-p^N) /xn 

= P^n+pDn- 

The claim follows by downward induction from the relations Di = tT~'^ Di^iT^^ , 
6 = tT-^^i+iT-_\, and Ei = tT'^E.+iT-^ . . 



Polynomials can be computed by induction from Mqjv, by the help of the Yang- 
Baxter graph ^20j : 



(9) M,,,, = Mj T, + -^^^ ] if v[i] < v[t + 1] 
and 

(10) = M,t{xn - 1), 

where [f [1], . . . , f [A^]]$ = [t>[2], . . . , t>[A^], t>[l] + 1] is the raising operator. 

Note these polynomials are nonhomogeneous and the spectral vector associated to 
equals Spec^. We will consider also the specialization = q"''^'"h^^'"^ M^. Alternatively, 
the Macdonald polynomial My can be defined (up a normalization) by interpolation: 

(11) M^{{u)) = for \v\ <\u\,uy^v 
and the coefficient of x'" is g""'*-^-'. 

The polynomial E^, can be recovered as a limit form from M^,: 

(12) E^(a;i, . . . ,xn) = lim a^'"^My{xi/a, . . . , xj^/a). 

a—^O 
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2.4. Symmetric Macdonald polynomials. Symmetric Macdonald polynomials are 
defined as the eigenfunctions of the symmetric polynomials in the variables They 
can be obtained by applying the symmetrizing operator 

(13) S^= J2Ta 

on the Macdonald polynomial Ex- where A~ is an increasing vector and = Ti-^ . . . Ti^ 
if cr = Si^ . . . Si^ is a shortest decomposition of a in elementary transpositions. The 
coefficient of in the expansion of Ex-Sn is the Poincare polynomial (pti&x) where 
©A denotes the stabilizer of A in ©tv- 

We will consider several normalizations of the symmetric Macdonald polynomial. First 
Pa is such that the coefficient of x'^ is 1. The normalization Pa = -fj^Px is interesting 
since the normalization reads: 



(14) Pa := ^ 



u+=X 

where denotes the unique nonincreasing vector which is a permutation of u. Finally, 
we will also use the specialization Jx = cx{q,t)Px with c^{q,t) = hg^t{v,t) which has 
integral coefficients when expanded in terms of monomials [T5] . 

For an infinite alphabet, symmetric Macdonald polynomials can be defined, up to multi- 
plication by a scalar, as the only basis having dominance properties which is orthogonal 
with respect to the scalar product 

m 1 A 

(15) (Pa, Pf^)q,t = zxll Y^^aT^^'/^' 

i=l 

where px = Vxi •••Pa„, Vk = Ylix&x^^ ^ power sum symmetric function for k > 
1) ^x,^l = 1 if A = /i and otherwise and zx = ni^i"^*'^"*' where rrij denotes the 
multiplicity of i in A. 

The dual basis of Pa is another normalization of Macdonald polynomials usually denoted 
by Qa. 

{Px, Q^i)q,t = Sx,^i■ 

The reproducing kernel 

^(^) 1 - 

K,,{X,Y) = J2Px{X)QxiY) = J^^^Hl r^^A(X)pA(Y), 

A A ^=l ^ ~ ^ ' 

admits a nice expression when stated in terms of the Cauchy function ai (X) = Ylxex 
and A-ring [T9] : 



(16) K{X,Y) = ai{- -XY^ ^ ' 



J aJ^XY 



where the alphabet jz^XY means {q^xy : i G N, x G X, y G Y} and j^XY means 
{tq'xy : i eN,x eX,y eY}. 
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Note formula fll6p remains valid for finite alphabets. 
Denote ca(X; q, t) = ^"^a'^ PaI^) the dual basis of px- We have 



-^(^' ^) = n r^^^(^) = " n S^^^(x)- 

i i 

For homogeneous polynomials R and 5 of global degree d we have 



Hence 

(17) PA(X;g-\ri) = ^PA(X;g,t). 



For the normalization Pa we have 



.-n(A) 

PA(X;g-\ri) = ^^^^^3^PA(X;g-\t-^). 




_1 



(_l)|A|^-|A|-n'(A)^-n(A)^/^^^^^) 



(18) PA(X;g-\ri) = ^ ^, " PA(X;g,t) = rAPA(X;g,t). 

Ca '-J 

with = (-l)IHg"'(^)t-"'(«+) (using the notation of Lascoux et al. [22]). 

2.5. Symmetric shifted Macdonald polynomials. Following the previous subsec- 
tion, we define the symmetric shifted Macdonald polynomials MS\ as the symmetriza- 
tion of Ma- , where v~ denotes the unique nondecreasing vector which is a permutation 
of f , such that the coefficient of equals More precisely, 

(19) Ma-5jv = 0i(6A)M5A, 
recall Sn is the symmetrizing operator f[T^ . 

We will use also the normalization MSa = ^ ! ! ' MS\. With such a normalization 
we have 

(20) MSa := 

n+=A 

Alternatively, the polynomials Ma are defined by interpolation: 

(21) M5A((/i)) = for |A| < |/i|,A ^ /i. 

Note that q^'^'^^^Px is the homogeneous component of maximal degree in MS\, that is 

(22) lim a^^^MSxixi/a, . . . , Xn/o) = q-'^'^^^Px- 

a— ^0 
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Equivalently, from equalitions (IT^ . ( |T^ and ( EUj) . we obtain 
(23) Fxixi, . . . ,xn) = lim a^^^MSx(xi/a, . . . ,xn/o)- 

a— >-0 

That is : PA(a^i, ■ ■ ■ iXn) is the homogeneous component of maximal degree in MSa- 

3. Generalized binomial coefficients 
3.1. Definitions. Sahi [53] generahzed binomial coefficients: 

(24) 



M.((^)) 
M,((t;)) 



'U 
-V 

From the vanishing properties we have 

r wi 

(25) = if \u\ < If I and v. 

More generally, a theorem of Knop [16] implies 

"7/1 

(26) =0iff+2M+, 



and in particular \qn\ = [^] = 1- 



Okounkov [27j and Lassalle [23] introduced independently the symmetric analogue of 
this coefficient: 

^27) MS,((f)) 



f y ■ Ms,((f)) 

3.2. Okounkov binomial formula and consequences. Symmetric binomial coeffi- 
cients appear in Okounkov's binomial formula. 
Define 

MS^Xi, ...,x^) = ^^^(^_i)|,| MSA(t^-^a;i, . . . , t^-^a;^; g-\ t"^). 
Theorems. (Okounkov) 

MSA(axi, . . . , ax^v) = E f MQlIm! ^^^(xi, . . . , x^) 

Note from eq. (l23ll and ( fTSl) . we obtain: 
(28) lim al^lMS'A(a;i/a, . . . , x^v/a) = ^"'(^^Pa. 

a— >-0 

Hence, if we apply Theorem [3] to the alphabet {xi/a, . . . ,xi^/a} and we take the limit 
when a tends to on the right hand side the we obtain: 

,29) MS.(.. .....„) ^ E []) i||^P.(-. ■ ■ ■ . 

The constant term M^,(0, . . . , 0) is related to the principal specialization of |22j : 
(30) M,(0,...,0) =r,E,((0)). 
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Note = Ty+ hence from eq. (ITl]) and (I2UI) we obtain 

MX) 

(31) MS,(0, . . . , 0) = r,P,((0)) = r,-^P,((0)). 

Hence, 



(32) MS.(..,...,..,^,"'(^.E^S|C) -..-.fMy^"'-- 



Or equivalently: 
Lemma 4. 



(33) 



MSx{xi,...,xn) = ( ^ I §77?vT^M(a;i,---,a;Ar). 

4. Principal specializations 



4.1. Principal specialization of E^. From equation ( !30|) the specialization £"^((0)) is 
obtained (up to a multiplicative constant {—l)*q*t°) from the constant term My{0, . . . , 0). 
For simplicity, we sometimes neglect the multiplication by (—1), q and t of our poly- 
nomials and denote it by (^). 

We introduce the classical g-Pochhammer symbol: {a,q)N = YliLii^ ~ '^Q'*"^) ^-iid its 
generalization for a partition A: 

N 

{a;q,t)x = Y[{at^~\q)x[i]. 

i=l 

Since ITj = t, we obtain 

(34) M..,^(0) = M.(0) ^ '^/^W < + 1] 

from (|9]). From f lTOj) we have also 

(35) M^$(0) = -M^(0). 
We have: 

Proposition 5. For any vector v G N^.' 

M„(o) = (-i)H (fY-');; 

Proof. It suffices to prove that the right hand side P{v) = ^ "^^ivlt) satisfies the 
recurrence relations f l34p and (135 p . 
Let us first prove that if v[i] < v[i + 1] 



(36) 



P{v) \ 1 - 
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Note only hgt{v,qt) changes in P(t>): 

P{v-Si) _ hg^t{v.Si,qt) 
P{v) hq^t{v,qt) 

The only factor in hq^t{v, qt) that changes is the factor for the cell (z + 1, + 1). So 

hg,t{v.Si,qt) _ 1 - 
hq^tiy,qt) ~ 1 - g^'ti+il-^Wt^ ' 

where 

a = As + 1, + 1) = J^-.s,(^ + l,v[i\ + 1) + 1. 

Let 

Ai = {I -A <i,v[l]>v [i]} U{l:l>i + l,v[i\<v [I]] , 
A2 = {I : I < i,v [I] > V [i + 1]} U {I : I > i + l,v [i + 1] < V [I]} , 

then A2 C Ai and 

Ai\A2 = {1:1 <i,v\i]<v[l]<v[i + l]} 

U{1 : l> i + l,v\i] <v[l] <v[i + l]}] 

thus a = #Ai\A2 + 1. Also [i] = jj^Ai + 2 and [i + I] = #-42 + 1, and so 
a = r^ [z] -r„ [i + 1]. This finishes the proof of ([36]) since ^^^jgjii = 

To prove the second part we need the following lemma 

Lemma 6. Let X be a decreasing partition and m < N the biggest integer such that 
X[m] > 0. Let 

7=[A[l],...,A[m-l],A[m]-l,0^-"^]. 
//M^(0) = P(7) then Ma(0) = P(A). 

Proof. Let 

/3 =[A[m]-l,A[l],...,A[m-l],0,...] 
a = [A [1] , . . . , A [m - 1] , 0, . . . 0, A [m]] . 

We evaluate (0) /M^ (0) , Ma (0) /M„ (0) and use (0) = -M^j (0) to prove Mx (0) /M^ (0) = 
P (A) /P(7). The formula ( l34l) is used repeatedly in the following calculations. First 
we have 

Ma (0) _ 1 - gAM^W-m+l 

Ma (0) ~ 1 - g^Mt 
Since A [m — 1] > A [m] it follows that 7 [i] > 7 [m] for 1 < i < m. Then 

Mi3 (0) _ ']pr^ 1 - g^H-^H+it^-i 

M^(0) ~ fj^ 1 _ gAW-AH + l^m-i+l- 

Next (^^ '^'^)a ^ Ci_o^MiAf+i-m\ g^^^ evaluate 1^' • In this ratio the 
only cells that have a changed factor are those in column A [m] and in row m. So 
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Jl A A [m]) = m — i and Jl^ (i, A [m]) = m — i — 1 ioi 1 < i < m. The cells in 
{(1, A [m]) , ■ ■ ■ (m — 1, A [m])} contribute 



m—1 

n I _ QX[i]-X[m]+lfm-i+l 

to the ratio and the cells in row m contribute 



Thus 

(gt^; g, t)^ /i,,^ (7, gt) _ 1 - t^-™+igAM ^ _ ^AW-AM+i^m-i 
{qt^;q,t)^hq,t{X,qt) ~ 1 - tg^H ^1 1 - gA[i]-AH+i^m-i+i • 

This agrees with — (0) /M^ (0) and proves the lemma. 



End of the proof of Proposition [5l Any vector v can be obtained from 0^ by 
adding 1 to a null part or to a minimal nonzero part. We will denote this operation by 

For instance, 

[0, 0, 0] 4[o, 1, 0] 4[o, 2, 0] 4[o, 3, 0] 3, 0]. 

Let V — v'. Suppose v'~^ = [Ai, . . . , A^, 0, . . . , 0] then t>+ = [Ai, . . . , A^-i, A^— 1, 0, . . . , 0]. 
Hence, if M^+(0, . . . , 0) = P{v^) then by lemma M M^,+ (0) = P{v'+). And using re- 
peatedly eq. diSD, we obtain M^(0) = P{v) implies M„/(0) = P{v'). 
Since Mqn{0) = P(O^) the result is shown by a straightforward induction. ■ 
As a direct consequence 

Corollary 7. 

(t^g;g,t)„+ 



hq^t{v,qt) 

Lascoux [21j gave an equivalent expression using infinite vectors 
v°^ = [v[l],...,v[N],v[l] + l,...,v[N] + l,...] 

and 

{vr = [{v)[l],...Av)[N],q{v)[l],...,q{v)[N],...]. 

Note 7^ {v°°). 
Lascoux showed 

(37) p(„)=(-i)Hjj n -^p^ 



i=l j>i {v)°°\j] 



- 1 
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4.2. Principal specialization of Px. From eq. ([3T]) and (1191) one has 

(38) Pa((0)) = WM5.(0) = W^^M,-(0). 

So one has to determine the value of the constant term Ma-(O). 

The t-multinomial coefficient is defined as follows: for any partition A G Nq let m\ (i) = 
#{j ■.\[j]=t} (defined for < ^ < A [1]) and let (^J^ = (t, t)^ / (^, ^)„.,« 
(note (t,t)^ = nr=i (1 ~ ^^))- is.ct, the Poincare series for A, denoted (©a), equals 
Y[i=o (^' ^)mx{i) I ^ ~t)^ where ©a is the stabilizer subgroup of A in ©at (the symmetric 
group on letters), and ^(©tv) = (t,t)N/{^ — t)^ . 

For application of proposition we find a convenient formula for Mx- (0) for parti- 
tions A. 

Proposition 8. For a partition A G Nq 



Mx- (0) = (-1) 



|a| (t^;g,t)^^(5A) 
hg^t{X,t) (t>{SN)' 



X (t^q- q t) 

Proof. From Proposition [5] Ma- (0) = (—1)'^' — / so we need to evaluate the 

"-g.t (A ,qtj 

{t^q;q,t) hg^t{X-,qt) 
ratios ^Yi^ \^ and — \ , ,s . it is easy to see that 



If the multiplicity of a particular A [i] is 1 (in A) then the leg-length of cell {N + 1 — i, j + 1) 
in A~ is the same as that of the cell («, j) in A for 1 < j < A To account for multi- 
plicities let c be the inverse of rx- , that is rx- [q] = i for 1 < i < A^. Then A~ [q] = A [i] 
(for example suppose A [1] > A [2] = A [3] > A [4], then ci = iV, ca = iV - 2, cg = - 1). 
As before 

Ja- (ci,j + l) = Ja(«,j),1 < J < 
The factor in h^^t (A",gt) at the cell (ci,j + 1) is ^1 - A(ij)+igA[i]-i^ vfhich is the 

/i fA~ ot) 

same as the factor at the cell (i,j) in hqt (A,t). Thus / ■, ' ,n is the product of 

the factors at the cells (q, 1) in hg^t{X~,qt) divided by the product of the factors at 
the cells (i, A[i]) in hq^t{X,t), for 1 < z < ^(A), since there are no cells for the zero 
parts. The factor at (q, 1) is(l — Suppose A [i] has multiplicity k (that is, 

A [i - 1] > \\i] = ... = \ [i + k-l] > \ [i + k]) then J a (i + / - 1, A \i]) = k - I ioi 
I < I < k, and the cells {i, A [i]) , . . . , {i + k — 1, \ [i]) contribute nf=i (^ ~ — i^i '^)k- 
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Thus 

Combining the ratios we find that 

hq,t [X'^qt] 

since (0) = - ^(A). 



1 

t 



As a direct consequence of Proposition |8] and equahty fl38l) . we obtain 
Corollary 9. 

hq,t (A,t) 



5. SUBRECTANGULAR MAGDONALD POLYNOMIALS 

5.1. The denominator of sub-rectangular Macdonald polynomials. The follow- 
ing lemma shows that if A C [m'^, 0^^^] with 2k < N then P\ does not have a pole at 
^m-i^N-k+i _ q^^^t 7^ 1 where > 1 is a common factor of m — 1 and 

N -k + l: 

Lemma 10. Let A C [m'^, 0^"^] he a partition with 2k < N. Then (1 - gm-i^N-k+i-^ 
not a factor of any the denominators of coefficients of x"" in P\. 

Proof. It suffices to prove that the integral version Jx does not vanish when 1 — 
^m-i^N-k+i _ g _ g'^^f 3 ^ Q "where d > 1 divides m — 1 and — A; + 1. In 

other words, we have to show that cx{q,t) does not have 1 — q^-^t^-^+^ as a factor. 
Suppose A = [mJ' , Xk'+i, . . . , \k",0^~'' ] with k' < k" < k and A^" > . Then the 
only factors with the relevant power of q that occur in c\ are (1 — g^' 

-i^A;-i+i) for 

\i = m (so i = l..k') and j = 1 (so A^- = k"). Hence, for such a pair {i,j) one has 
X'j — i + l<k<N — k. This proves the result. h 



16 



5.2. The (g, t)-binomial 
binomial ( ™ ] for A C m^: 



(39) 



m 
A 
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j . Lassalle [23] gave an explicit formula for the (g, t)- 



(i,i)eA 



-A; 



1-g 



1 



Using (g, t)-hook products, the formula reads 

(40) 
We deduce 



A ^ ^ ^ /.,,(A,g)/.,,(A,t) • 



Lemma 11. Let A C m'^' be a strict suhrectangular partition with 2k < N. The denom- 
inator of the reduced fraction 



m-lj.N-k+1 



has no factor 1 — g™ t 



Proof. We examine the factor of the denominator of equality ( |39|) : 



/ 



D.:= n 

(ij)6A 



1-g 



V 



1-g 



1+ 



V 



and 1 - q*t^'i * in 



Since A is a partition, there are only terms of the type 1 — g*^'^.'"*'''^ 
Dx- When A C m^, we have X'j — i + 1 < X'j + 1 < k + 1. And so k < N — k implies 
A^ — z + l<A^ — + This proves the result. ■ 



Note 

5.3. Principal specialization for suhrectangular partitions. We consider certain 
poles of Ma- (0) for A C [m'',0^-''] with 2k < N. In particular, examine the oc- 
currences of (l — g'^^^t^"'^"'"^) in (t^;g,t)^ and hq^t{X,t). Since the maximum leg- 
length of any cell in A C [m'^, O^"'^] is — 1 only factors of the form (l — t"g'') with 
l<a<k<N — k + 1 and < 6 < m — 1 can appear in hg^t (A, t). Further 

(*"i''.').=rin(i-'"""v-'). 

1=1 j=i 

The factor (l - qn^-ifN-k+i^ 

appears in this product exactly when A [k] = m, that is, 
A = [m^ O^-'^] . Thus M^- (0) has a zero at g^-^t^-k+i = i for A = [m^ O^-'^] and no 
zeros or poles there when A C |^m*^, O^"'^] . We summarize this in the following lemma 
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Lemma 12. Let A C [m'^jO^^^] with 2k < N then the reduced fractions Mx-{0) and 
Px((0)) have (l - as « factor if and only if X = [m^, 0^"'']. 

Equivalently, if (1 — g™--i;(;^-fc+i) = g and for any d > 1 dividing m — 1 and N — k + 1, 
[1 — q d t d Jt^O then 

^((0)) _n 
^a((0)) • 

Note we can write another proof using eq. (1371) (see appendix [B|) . 

6. Rectangular Macdonald polynomials 

6.1. Clustering properties of MS^k. In this section, we will use finite alphabets 
with different sizes. 

Notations 13. For simplicity, when we use a partition A G , we mean that the 
underlying alphabet is {xi, . . . ,X]y} and the symmetric shifted monic Macdonald poly- 
nomial will be denoted by MS^^ = MS\{xi, . . . , x^). In the same way, for v G the 

nonsymmetric shifted monic Macdonald polynomial will be denoted M^^-* . 
The length of v ^ is i{v) := ma.x{i : v[i]> 0}. For k > i{v) we will denote also 
:= [v-i^^ ...,Vk]. 

The following proposition shows that if f G with i{v) < k then the Macdonald 
polynomials Mv^^ and M^'^}^ are clearly related. 

Proposition 14. The following assertions hold: 

(1) Suppose V G Nq satisfies i{v) < k for some k < N , then 

. . . , . . . , t, i) = t^^-'^^HM^g . 

(2) Suppose a partition A G satisfies £ (A) < k for some k < N, then 

M^f ^ . . . , t^-'^-s . . . , t, i) = t(^"'^)i^iM5f (x) . 

Proof. We prove first (1): Both sides have the same coefficient of x"" and satisfy the 
same vanishing conditions. More precisely, the both sides vanish for x = (u), u & 
v\,u 7^ V, and {u) [j] = q^^h^ ^"W, for 1 < j < k; and the multiplicative 
coefficient is obtained considering the coefficient of the dominant term in the left hand 
side and the right hand side. 

To prove (2) remark the left-hand side is symmetric in (xi, . . . ,Xk)- Hence an obvious 
modification of the above argument applies. ■ 



Example 15. For instance consider v = [2,0]. The polynomial M[2,o](a;i, X2) vanishes 
for [x,,X2] = (00) = [t,l], [x,,X2] = (10) = [qt,l], [x^x^] = (01) = [l,qt], [xi,X2] = 
(11) = [qt,q], [xi,X2] = (02) = [l,^^^] whilst M[2o]((20)) ^ (recall that (20) = [qH,l]). 
Hence, the polynomial P{xi) = M[2,o](a;it, 1) vanishes for [xi] = [1] = (0) and [xi] = 
[q] = (1) whilst P((2)) = M[2,o](g^t, 1) ^ 0. This agrees with the definition of M[2](xi) 
up to a multiplicative factor. 
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For almost rectangular partitions Vm,k '■= [m'^'^ , (m + 1)^] we prove that the corre- 
sponding Macdonald polynomials nicely factorize: 

Proposition 16. For m >Q and < k < N we have 

N-k N 
1=1 i=N-k+l 

Proof. Recall u.^ ^ [u[2] , . . . ,u[N - 1] ,u[l] + 1], thus Vm,k-^ = Vm,k+i ior < k < N 
and f„ 7v_i.$ = Vm+1,0- The claim is obvious for vo,q- Suppose the claim is true for 
some {m,k), then 

7V-fc-l N-1 
i=l i=N-k 

Multiply both sides by [xn — 1) to get an expression for M^f^j..^^ (or Vm+i,o)'i observe 
(xjv — 1) {q~^XN, q~^)jn — — {xn, Q^^)m+v This completes the inductive proof. ^ 



As a straightforward consequence, setting A; = in the previous proposition, My^ ^ is 
symmetric in xi, . . . ,xn^ and then we find: 

Corollary 17. For n > 1 MSi% = gM«-i)/2 (-l)'^"nti i^uq-^)^; note Vn,o = [n''] . 

Let X = {xi, . . . , Xn} and Y = {yi, . . . , i/k} be two alphabets. We will denote by 
i?(X, Y) = Y[f=i Y[j=ii^i ~ Vj) resultant of X with respect to Y. 
Let also 

V%^^^^{x,, ...,Xk):= R{[x,, . . .,Xki . . . 

We have: 

Proposition 18. Let A = [m*^, 0^"*^] for some 1 < k < N and m >1, and y then 

M5f ) . . . , . . . , i) = y-'-v%,kJ^„ . . .,.,). 

Proof. By the previous propositions 
and 

m— 1 

= (-1)- ^-m(m-l)/2^-m(iV-fe)^-m _ ^^Ar-^^^,-^ ^ 

j=0 

for 1 < i < /c. ■ 
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6.2. Homogeneity of MS^k at 1 - qr^-H^-k+^ = 0. 

Let us describe first the solution set Z of tlie conditions 

for any common divisor a of m — 1 and N — k + 1 witli a > 1 . 
Let d:=gcd{m-l,N -k + l),mo = Uq = 
Recall the Euler 0-function, cf) (d) — where 

Kd:^{k:l<k<d, gcd {k, d) ^ 1} . 

We claim that Z has (p{d) disjoint connected components as a subset of C^. Let 
z = re^""'^ e C with r > and ^ e R, and set 

q = z''°,t^ujz-"'°, 

then q'^otno ^ ^no ^^^j jg squired that u;"*"^ = 1 and (a;"")-' 7^ 1 for 1 < j < d. 
(if (i = 1 the latter restriction becomes void). For determining connected compo- 
nents it suffices to let r = 1 and project onto the 2-torus realized as the unit square 
{{en, (3) : < a, /3 < 1} with identifications (a + 1, /3) = (a, /3) and (a, (3 + 1) = {a, (3), 
with the mapping {a,/3) 1 — )■ (e^'^*", e^'"*'^) . The pre-image of (g, t) is 

{{noe,-moe + ij) -.eeR}. 

We require ^n^d E Z and V^noj ^ Z for 1 < J < c?, that is ^ = for some 
/c and ^ ^ Z (when d = 1 the value /c = is permissible). This is equivalent 
to gcd {k,d) — 1. By the identification assume < -0 < 1. So there are 
solutions for ^0 where K :— {k:l<k< nod, gcd (/c, rf) = 1}, and H^K — no(f){d). 
For k e K let Z^ ^ ^ (no9, -moO + : < 61 < ^ j. Then Zk meets Zk' where 

k' — {k — mod) mod (no(i) (the first coordinate is periodic for ^ i->- ^-|- ^; and where 
< a mod 6<6foraeZ,6>l). This is continued to show that Z^ is connected to 
Z^i with k' — {k — rriodj) mod (rioo?) , 1 < j < no. Consider the set 

K' :— {{k — ruodj) mod {nod) ,0 < j < no,k E Kd} ■ 

We claim this set coincides with K. If k, k' e Kd and k ^ k' then (/c — modj) ^ 
{k' — modj') mod {nod) for any j, j' (else k = k' mod d); suppose < j < j' < no then 
(fc — modj) ^ {k — m,()dj') mod (norf) or else mod {j — j') = nodi for some /, which is 
impossible for I < j — j' < no- Thus K' G K and ^^K' = and hence K' = K. 
Thus Z is the disjoint union of the connected sets [J {Z^^jd '■ < j < no} x {r : r > 0} 
for k e Kd. 

Example 19. (1) First examine the solution of qH^ — 1 where gcd (a, 6) = 1 on the 

example a — 4 and b = 7. Topologically we can collapse the problem to the 
torus [q — exp{2iTTa),t — exp(2i7r/3)] , < a, (3 < 1. On the unit square this 
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becomes 7a + 4/3 = mod 1, or /3 
see figJH 



-ja, interpreted periodically, period 1 
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Figure 1 


. Solution of 




-- 1 for 


a = 4 and 6 = 7. 



(2) When gcd(a, b) > 1, for instance a = 8 and b = 12, there are several connected 
solutions. In the example given in figEl there are two solutions 2a + 3/3 = | 
and 2a + 3/3 = |. 

Proposition 20. Consider q = z"" and t = uz^ a specialization of the parameters such 
that 1 - q^-H^-k+i = andl- q'^t^^^ if d > 1 divides m - 1 and N - k + 1. 
Then, 



Proof. From Lemma HJ one has 

MS^k = P^.+ '£—{'^') 



From Lemma fT2l ^p"'}/^"^^'' = for yU C m^. Furthermore, from Lemmas fTOl and fTTl there 



(otherwise the denominators of 



are no poles at g = 2;" and t = coz^ in and ^™ 

the two polynomials have (1 — g^-i^^-'^^+i) as a factor which contradicts the lemmas). 
The result follows. ■ 
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Figure 2. Solution of = 1 for a = 12 and 6 = 8. 

7. Proof of a conjecture of Forrester 

7.1. Clustering properties of P^*- the study of fractional quantum Hall states, 
Bernevig and Haldane ^ identified certain clustering conditions on rectangular Jack 
polynomials. These can be interpreted as factorization properties under certain special- 
ization of the variables. In this context, Baratta and Forrester conjectured [2] a more 
general identity involving a rectangular Macdonald polynomial: 

(42) P^k{y,yq^, . . . ,yq^^^^ ,XN-k, ...,Xn) = T^N^^ixi, . . .,Xk), 

where N >2k,a = -^^^zj^ and gcd(A^ - + 1, m - 1) = 1. 
Propositions [20] and [18] allow us to write a more general formula. 

Theorem 21. Let ?/ G N. Consider q = z"' and t = ujz^ a specialization of the 
parameters such that 1 — q'^~^t^-k+^ = q and 1 — q^^t a ^ o z/ (i > 1 divides 
m — 1 and N — k + 1. Then, 

P^k{xu . . . ,Xk,yt^~^~^, ...,yt,y) = V%^.^^{xu . . .,Xk). 

Proof. Propositions UU\ and [T5] shows the formula for y y^O. Since the coefficients of x^ 
in the left hand side and right hand side are polynomial in y, the result remains true 
for y = 0. ■ 
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Example 22. Let us illustrate this result with few examples: 

(1) Set g = t-^ then 

y y 

P%2flfi]{.xi,X2,yt,y) = (xi - j){x2 - -)(xi -yt'^){x2 - yt^) 

(2) Let t = —q^^, one has: 

y 

Pmixi,y) = {xi + -){xi + y){xi + qy). 
q 

whilst the result does not hold for t = g^^: 

^[3,0] {xi ,y) = ^1 + yx-^ + y'^^ (xi + y). 

(3) Let u = exp {^} and t = uq~^. We have 

^[4,4,o,o](a;i,a;2,t?/,?/) = {xi - U}^yq-'^){x2 - uj'^yq~'^){xi - u'^yq-^){x2 - uj'^yq~^)x 

x(xi - uj'^y){x2 - uj^y){xi - uj'^yq){x2 - OJ^yq), 

whilst the result does not hold for t = g~^, since there is a pole in P[4,4,o,o]: 

Nnmei{P[i^4,ofi{xi,X2,ty,y))\t=g-i = 3— (1 - - - q^)xlxl, 

where Numer denotes the numerator of the reduced fraction. 

( N-k + l \ 

7.2. The Jack polynomial P^^.^""" . The Jack polynomial is usually recovered 
from the Macdonald polynomial P\{xi, . . . ,xn', q,t) by setting q = t"" and taking the 
limit t — )■ 1. 

So as a consequence of Theorem [2T] we obtain 

Corollary 23. LetN, k >1 andm > 2 verifying N > 2k and gcd{N—k+l,m—l) = 1. 
Then we have 



i=l 

See Kakei et al. pH] for related results on shifted Jack polynomials. 

7.3. Clustering properties of nonsymmetric Macdonald polynomials. As be- 
fore we use (*) to indicate a term with a, 6 G Z when its value can be ignored in the 
context of analyzing zeros and poles. We restate a result of Lascoux et al. [22| Corollary 
4.3 from Sahi's binomial formula]. For m G let := {u G : [^] ^0,v ^u}. As 
stated before v E Bu implies C and \v\ < \u\, but there are important additional 
consequences as we will show. 



Theorem 24. For u eNq 



(x) = (x) + 5^ 



E^ ix) . 
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We need to use this formula for u = ^m^,0^ ^] with 2k < N and {q,t) in a neigh- 
borhood Q of the set of solutions of q^t^-^+^ = l and qH'' ^ 1 for 1 < a < m and 

< b < N — k + 1, except for a = m and b = N — k + 1. (We note here that if 
g, t G C and neither q nor t are roots of unity then q^t^~^+'^ = l and = 1 imply 
that a = ^m, b = ^ {N — k + 1) where d = gcd (m, N — k + 1) and n E Z; roughly 
the equations imply q^-'^t^"^ = 1 = q<imfa(N-k+i) ^ ^^^ -^ From Knop \16\ Prop. 5.3] the 
coefficients of h^ t (t>, qt) (x) are in Z [q, q~^, t, t~^] for any v G Nq (the same multi- 
plier works for (x)). Furthermore {{v)) = (*) h^ t {v,q) (from Lascoux et a/.[22| 

(t^q,q,t) 

p. 8]), and i^^, ((0)) = (*) ^ J^J ^^"^^ by Proposition O Thus we consider hg t iy.qt) and 

hq^t {v, q) for v G -B^. We showed that <Z u and < \u\ implies (t^g, g, t) + 7^ for 
{q,t)e^. 

Proposition 25. Suppose u G anc? £ (n) < k < N then v E Bu implies i (v) < k. 

Proof. We use the "extra- vanishing" theorem of Knop [121 Thm. 4.5] applied to v with 
My {{u)) 7^ 0. This implies that there is a permutation w E &n such that for any i with 

1 < i < the inequality i >w {i) implies v[i] <u [w {i)] otherwise v[i] < u [w {%)]. By 
hypothesis u [j] = for any j > k hence (j) > j (since v [w~^ (j)] < is impossible). 
Thus (j) > j for each j > k and by downward induction w (j) = j for k < j < N. 
By applying Knop's condition it follows that v [j] < u[w (j)] = u [j] = for j > k. ^ 



Corollary 26. If u = [m'',0^ '^j and v E then hq^t{v,qt) 7^ and hq^t{v,q) 7^ 
for {q,t) E n. 

Proof. The factors in the hook products are of the form 1 — g'^t^ with 1 < a < m and 
0<b<k<N — k + 1 (the leg-lengths are bounded by A; — 1). h 



This shows that has no poles for v E B^k and (g, t) E The last step is 

to show E^,k ((0)) = for g'^t^-^+i = 1 and (g,t) G Q, and indeed 

k m 

i=i j=i 

the factor with i = k, j = m is (l — q^t^^^^^^. 

Proposition 27. Suppose u= [m\0^-^] ,2k <N, q"^t^-^+^ = 1 andq"'^H^^-^+^^^'' 7^ 
1 /or a > 1 6emg' a common divisor of m and N — k + 1, then 



M„, (x) 



Eu{xu...,Xk,yt^ ^^^...,yt,y) 
Proof. The proof is essentially the same as that of Proposition 
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We find that the polynomial (x) is singular for certain values of (g, t) exactly when 
it coincides with M„ (x), and in that case there is a factorization result of clustering 
type. 

7.4. Highest weight and singular Macdonald polynomials. In the case of special 
parameter values when MSx = P\ by Proposition [2] it follows that MSx J2iLi '^i = 
P\ J2i=i '^i — -^A Si=i 6 and thus Pa Si=i = 0. For rectangular partitions, this 
matches with a result of one of the authors (J.-G.L.) with Th. Jolicoeur [12] which 
involves the kernel of the operator 



^^^^^ rf> /y» /~k ry 

j=i j=i -^j ^g-^* 



with 



d f{xi,...,XN)-f{xi,...,Xi-i,qXi,Xi+i,...XN) 

-f{Xi,...,XN) - 



dqXi Xi QXi 

Note from a result of Baker and Forrester [U eq. (5.9) pl2], the operator L+ satisfies 

N 

(l-g)L+ = ^A. 



1=1 



Theorem 28. (Jolicoeur-LuqueJflE] 

If N > 2k then the Macdonald polynomial Pmk{xi, . . . ,Xn', (l,t) belongs to the kernel of 
for the specialization {q,t) = (^z'' ^ ^ , cjz^^ j where d = gcd{m — l, N — k — 1) and 

u = exp I^^^^^TTY^I 'with n E Z,. 

We observe similar phenomena for nonsymmetric Macdonald polynomials: Note 
Eu, Mu have the same eigenvalues under the action of ^t, Sj respectively. If for certain 
parameters (g, t) M„ = Eu then EuDi = for 1 < z < (since M^jSj = Eu'^i = 
Euii + EuDi). 

According to Proposition |271 Suppose u = [m'^, O^^'^] 2k < N and g^^^-^+i = i 
and ^ 1 for a > 1 being a common divisor of m and — A; + 1, then 

E^kQN-kDi = for each 1 < z < A^. 

8. Conclusion and perspectives 

One of the authors (J.-G.L.) with Thierry Jolicoeur[T8] investigated other families 
of symmetric Macdonald polynomials belonging to the kernel of L+ for certain spe- 
cializations of the parameters. These polynomials are indexed by staircase partitions 
and numerical evidence shows that they have nice factorization properties. A staircase 
partition A = [((/3 + l)s + r)*^, {13s + rY, . . . , (s + r)^, 0^] is defined by five integer 
parameters /3, s, r, k and i such that k < i and A^ = j^r + i{f3 + 1) + k E N. The 

corresponding specialization is (t, q) = {z~ , z s~u;) where g = gcd(£ + 1, s — 1) and 
oj = exp I ^^^(^"^'^g) I _ The simplest polynomials {r = k = 0) can be factorized for any 
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variables (g, t)-discrimmant. When g = 1, this is a consequence of a 

theorem proved by one of the authors (J.-G.L) with A. Boussicault |S1 Theorem 3.2]. 
For instance, 

Pi2o{xi,X2,X3;q = r'^,t) = {-)t{xi-tx2)ixi-txs){x2-txi){x2-tx3){xs-txi){xs-txi), 

where {—)t denotes a factor depending only on t. But it seems that a factorization 
holds also for the other cases (for special values of the variables and the parameters). 
Let us give a few examples: 

P63o{Xl,X2,X3;q = -t-^,t) = {-)t{Xl +X2){Xi + X3){X2 + X3)(xi - tX2){Xi - tx^) 

{X2 - tXi){x2 - tX3){Xs - tXi){x3 - tXi), 

Pbsoooixi, X2, y, yt, yt^; q = f'^^i) = (-)t(xi - yt^){xi - yt){xi - yt~^) 

{x2 - yt^){x2 - yt){x2 - yt-^), 

P422ooixi,yi,yit,y2,y2t;q = ^"^^) = i-)tixi - yit^){xi - y2t^){,xi - yit){xi - y2t) 

{yi - ty2){yi - t^y2){y2 - tyi){y2 - t^yi), 

-P64ooooo(a;l,^/l,l/lt,?/lt^l/2,Z/2^,l/2^^g = = {-)t{xi -yit^){xi -yit^){xi - yit'^) 

(xi - y2t^){,xi - y2t^)ixi - y2t' ){yi - ty2){yi - t^y2){,y2 - tyi){y2 - t^yi), 

P750-^{xi,yi, . . .,yit^,y2, . . .,y2t^;q = t^,t) = (-)t(yi - Xit^){yi - y2t^)iyi - Xit^) 

{yi - y2t^){y2 - xit^){y2 - yit^)Pi2o{xi,yi,y2;q = t''^,t), . . . 
A correct formula (and of course a proof) remains to be found. 

Note there are also analogous formulas for singular nonsymmetric Macdonald polyno- 
mials indexed by staircase partitions. Consider the following examples: 

E2io{Xi,X2,X3;q = Z-'^,t = z) = i-)titX2 - Xi){tX3 - Xi){tX3 - X2) 

Eqso{xi,X2, X3; q = Z-'^,t = Z^) = (-)^ {X2 Z - X3) {-ZX3 + X2) ix2 - Z^X3) (Xi Z - X3) 

{-ZX3 + Xi) (Xi - Z^X3) (Xi Z - X2) (Xi - X2 Z) {Xi - Xg Z^) , 

E420 (Xi , X2, X3; g = -t, t) = {-)^{x2 + X3) (-tX3 + X2) (X3 + Xi) (-tX3 + Xi) (Xi + X2) (xi - 
X2t), 

E22nooixi,X2,yutyi,y2,ty2;q = t"^^) = (-)t(2/i - y2t^){.yi - y2t)ix2 - y2t^) 

(x2 - t^yi)ixi - y2t^){xi - t^yi), 

^4422oo(a;l,l/l,^/l^^^/2,l/22^g = = z"^) = i-)ziyi - y2z'^){yi - y2z){yiz - 1/2) 

{yi - y2Z^){xi - y2Z^)ixi - y2z){xi - yiz'^){xi - yiz). 

There are also equations involving permutations of (m'^,0^"'^). For instance: 

-Eo022(t, 1,3:3, 3:4; q = t~^,t) = {-)t{tX4 - y)(tX3 - ?/)x4X3. 

In appendix [C] we show a factorization formula for permutations of m^0^~^. 
The connections with singular properties remain to be investigated. In particular, if 
is singular then it has the same spectral vector for {Hj} as M^. If the spectral vector is 
nondegenerate (multiplicity = 1) then M„ = E^- 
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Appendix A. About notations 

For parameters a, b, c, we consider the operator ^ ^ acting on polynomials (or Lau- 
rent polynomials) by 

G:,, = a + " (1 - , 1 < . < AT. 

The braid relations G^G'+^G* = G^'^^G^G^'^^ hold for two families of solutions: 
(a, b, c) = {ti, —ti, t2) and (a, 6, c) = (ti, t2, — ti). In both cases the quadratic relation 

{G\^b,c ~ ^l) {G'a,b,c - ^2) = 

holds. The inverses are 

Some pertinent evaluations are 1G\^^^ = a, XiGl^_-i-_^^f^ = —t2Xi^i, Xj+iGJ^^^^.-ti = —hxi- 
In our applications we require the quadratic relation (G^ bc^ ^) (^a 6 c + l) =0? ^^d 
the evaluation IG^^^ = t, because 1 is (trivially) symmetric, thus a = t,b = —1. The 
version of Ti used in one of our previous papers [9] is Gl _f _i, but Lascoux et al. [22j 

uses G\ _l _^ which equals {g\h (that is, Tj (1/t)^^ from our notations[9]). 

With parameters 7, ^ in relation to our notations [9j the following hold for the Lascoux 
et al. [22] versions (and % = G\ _^ T^^ = Gi/t-i/t-i )'■ 

(1) ^. = t^-%_^ . . . TitT^I^ ■■■Tr^ (Cherednik operator) 

(2) pt{x) =p{xN/q,Xi,...,XN-i); 

(3) Djy = {1 — ^7v) a^Tv^; a simple check for the coefficient of ^n- apply the operator 
to the constant polynomial 1, since ITj = t we have = and it is necessary 
that IDn = 0. 

(4) A = tTr^D,^,Tr^ 

(5) Sj = t-^~*Tj_i . . . TiT ^1 — ^ j ^Afii • • • + similarly check the constants 
by verifying lEi = f'^; note {j^^ Tr^ = j-- 

Appendix B. Alternative proof of Lemma [12] 

Let A C [m^0^-^•] with 2k < N. We have A = [m^', A^z+i, . . . , Afe", O^-'^"] with 
k' < k" < k and A^'+i < m. 
The vector (A^)°° with A C is 

\j.N-k"-l + 1 ^ ^ ^m+N-l ^m+N-k' ^+N-k"-l ^± ^ 1 

So we have to enumerate the pairs with i = 1 . . .k' and j = 1 . . . N — k" verifying 

{>^~)N-k'+i = q'^t''-\ (A-)^+^. = gt^-'^"-^' and ^"^^^f^"'^' = g™-^t^-'=+^ Then, we 
have to enumerate the pairs with i = 1 . . . k' , j = 1 . . . N — k" such that N — i — 
{N — k" — j) = N — k + 1. Equivalently, j = N — k — k" + 1 + i. Hence, when i ranges 
l...k',j ranges N - k - k" + 2 . . . N - k + k' - k" + 1. But since k < N-k and k" < k 
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we have 2 < k - k" + 2 < N - k - k" + 2. If A C [m*-', 0^"'^'] then k' < k and we have 
N -k + k' -k" <N - k". 

To resume: when A C [m'^, O^"'^], j ranges N — k — k" + 2...N — k + k' — k' + l C 
1 . . . - k". So there aie N - k + k' - k" + I - {N - k - k" + 2) + I = k' pairs (i, j) 
satisfying the property. It follows that the maximal power of (1 — g^-i^^-^^+i) in the 
denominator of equality fl37|) is k' . 

But if A = [m^, then j ranges N — 2k + 2...N — k {k = k' = k"); so the maximal 

power is fc' — 1 = A; — 1. 

Similarly, we study the numerator: we have to enumerate the pair with i = 1 . . . k', 
j = I...N - k" such that N - i - {N - k" - j) + I = N - k + 1. Equivalently, 
j = N — k — k" +i. Hence, when i ranges 1 . . .k', j ranges N — k — k" + 1 . . . N — k + k' — k". 
Hence, when i ranges 1 . . .k', j ranges N — k — k" + l...N — k + k' — k" . Again 
N-k-k" + l...N-k + k'-k"(Zl...N- k" . So there are - A; + A;' - k" - {N - 
k — k" + 1) + 1 = k' pairs («, j) satisfying the property. As a consequence the maximal 
power of (1 — in the numerator of (!37|) equals k' . 

If A C [m'^', 0^^^], the numerator and the denominator simplify and we have no factor 
of the form (1 — g^-i^^-'^+i) in (|37|) . On the other hand, when A = [m''', 0^"*^], after 
simplification it remains a factor (1 — g'^-i^^-'^+i) in f |37p . This completes the proof. 

Appendix C. Expressions for M„ where m is a permutation of [m'^O^"''] 

Fix m > 1 and k with 1 < A; < A^. As before we neglect scalar multipliers (*) of the 
form g^t'', and then adjust the formulae using the monic properties. Define 

n^^k := {/3 G N'^ : 1 < /3 [1] < /3 [2] < . . . < /3 [A;] < AT} 

For /3 e VLN,k define u {(3) e by u [/3 [i]] = m for 1 < z < A; otherwise u {(3) [j] = 0. 
Let a denote the set : I < i < k}. For 1 < i < k define Ei by Si [j] = 6ij. 

Remark 29. Note if i = — 1 ^ a{l3) for some j then u{[5).Si = u{(5 — Sj). For 
instance, if A^ = 7 and [3 = [1, 4, 5, 7] then u{f3) = [m, 0, 0, m, m, 0, m] and 

u{(3)sq = [m, 0, 0, m, m, m, 0] = u{[l, 4, 5, 6]) = u{f3 — £4). 

Definition 30. For 1 < i < A^ /ei X/3 (0 = ^ - ^ - # : W > i}> (^f i = P [^] then 
X(3 (i) = N — i — k + I). Also let x^^^ be the point given by 

^ f Xi,ie(r{(3) 

The meaning of xp is simply that X/3 (0 = N — ru{p) [i\ and xf^ = {u{(3)) [i] for 
i ^ cr The following is one of the main results of this section. The proof consists of 
several lemmas. 

Theorem 31. For (3 e QN,k 

k m—1 

j=l i=l 



28 CHARLES F. DUNKL AND JEAN-GABRIEL LUQUE 

Lemma 32. Suppose v G N^, v[i] < v[i + 1] for some i < N and y is a point such 
that My {y) =0, then 

M.,, {y) = (yi^^) My (y.s,) . 
Proof. This follows from Definition [T] of Tj. h 



Lemma 33. Suppose (3 G VLj^^k and f3[j] — 1 ^ a for some j . Set i = (3[j] — 1 so 
that u [i] = 0, M [i + 1]= m and u .Sj = u{f3 — Ej) (see Remark\2§i). Then 

M„(^) {x^^\si) =0. 

Proof. Let G satisfy f [/] > 1 for / G (x (/3 - e^) and [/] = for / ^ a (/3 - e^) 
(in particular f [i] > 1 and f + 1] = 0). By Knop's "extra- vanishing" theorem 
Thm. 4.5] Mu{i3) ((f)) = 0. To prove by contradiction suppose Mu{i3) ((f)) ^ then 
there is a permutation w G ©tv such that for any i with 1 < i < either i > w (i) and 
u (/3) [i] < t; [w (i)] or i < w (i) and m (/3) [i] < w [w («)]. Let = {1, . . . , iV} \a (/3) and 
^2 = {1, . . . , N} \a {f3 — Ej). The sets 5*1 and ^2 agree with the exception i G 5'i\S'2 
and i + 1 G S'2\S'i. By construction w {I) G ^2 implies / G Si and / > w (/). Thus w 
maps 5*1 onto 5*2 and by induction (using I >w {I)) we see that w (l) = I for each / G 5*1 
with I < i. But then w {i) > i + 1, which is impossible. 
Consider the vector (v); by construction 

t^0^^\l ^a{/3 -Ej) 

Note 1 < [I] < k for I E a {f3 — ej), and (v) = x^^\si where each xi is of the form g"t^ 
with a>landA^-A; + l<6<A^. Now M„ {x^^\si) is a polynomial in k variables of 
degree mk which vanishes at infinitely many points of this form; by the uniqueness of 
the vanishing property of shifted Macdonald polynomials M„ (^x^^\sij = for all x^'^\ 



Lemma 34. Suppose 13= [N — k + 1, . . . , N] then 

x(^) = ^t''-'+\...,t,l,XN-k+l,---,XM) 

and 

N m~l 

Mu(p) (x^'^)) = n - 1) n (^^ - ^'^~'^') • 

i=N-k+l j=l 



(v) [I] 



Proof. Let v 



(m — 1)^ , 0^ '^J . By the results of Section 16.11 

k m—2 

M„ (xi, . . . , xk, . . . , 1) = n n (^^ - ^""''i') 

i=l j=0 
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if m = 1 the product equals 1. From v.^'' = u it follows that 

N 

Mu{(3)ix) = {*)M^{xN-k+i/q,---,XN/q,Xi,...,XN-k) Yi i^i-'^)- 

i=N-k+l 

Set Xi = t^-f'-i ioil<i<N -k, then 

Mu(p) {x) = {*) {xN-k+i/q, xn/q, t^^'"'^, ■ ■ ■ ,l) Ylf=N-k+i i^i - 1) 

= (*) nf=^_.+i - 1) uT=o i^^/Q - t''-'<i') ■ 

This proves the Lemma. h 



For (3 G l^AT^A: define 

Fp {x) := n i^m - . 
The special cases are (3 = [A^ — A; + 1, . . . , A^], = YliLN-k+i ~ 1) P ~ 

[i,2,...,fc],F, = nli(x.-t^-'=). 

Proof, (of Theorem |3T]): We use induction for steps of the form u — )■ u .Sj applied 
to u with u [i] = and u + 1] = m. The previous Lemma provides the 
starting point. Suppose for some j, i that f3[j] = i + 1 and i ^ a then with the 
identification Xi = Xj+i we have x'^^\si = x^^'\ where l3' = (5 — Sj. Then 

/ \ T' — +N+j-k-i 

because only the factor involving Xi changes. Suppose that the claimed formula holds 
for /3, that is, 

m— 1 

By Lemma [33] ^m(/3) (a^'-'^^-Sj) = and by Lemma [32] 

because x^^'^.s^ = x^^\ Also a;^^'^ = xf^ = where X/3 (0 = N-i-# {I : p [I] > i} = 
N -i-{k-j + 1). Thus 

and this proves the formula for P — ej. h 
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Example 35. Let (3 = [2,5,6,9] E (^io,4, then u {(3) = [OmOOmmOOmO], 

= {t^, X2, t^, t^, X5, X6, t^ t, Xg, 1) , 
Muil3) (X^^^) = ix2 - t') (X5 - t^) (X6 - t^) (xq - t) 

^ nje{2,5,6,9} ni=i i^j ~ t^^i'') ■ 

The factorization result for rectangular Eu can be adapted to this situation. Start 
with u= [m'', 0^-''] with 2k < N. 

For the rest of this section assume g^t^^^+i = i and no relation q°'t^ = 1 with a < m 
OTb<N — k + 1 holds (for details see Section [6l2|) . 

Then by Proposition [27] M„ (x) = Eu (x). We claim this equality can be extended to 
u {/3) provided that m is a reverse lattice permutation; this means that any substring 
[u [j] , u [j + 1] , . . . ,u [A^]] contains at least as many O's as m's [Ml p.313]. 
This condition is equivalent to 

(43) /3 [j] <N-2k + 2j-l,l<J<k. 

As above suppose (3 [j] = i + 1, (3 [j — 1] < i and v = u (/3) , v.Si = u{f3 — Sj). The 
inverse relation is 

(tC-l)(C-t) '^■H^ 

where C, = q"^t^+^+^-'^i _ The same transformation takes Ey si to E^. We use induction. 
Suppose Mu{i3-ej) = Eu{i3-ej) wheu q™'t^-^+^ = i and both polynomials are defined 
(that is, the coefficients have no poles at q^t^-^+'^ = i). Then the same properties 
hold for M„(;3) and Eu(,3) if C t-\l,t. Set = t-C^-^+i) then C = with a = 
2k + i — 2j — N. But i = f3[j] — 1 thus by condition (143!) we have 

a <2k -2j - N + {N -2k + 2j - 1) - 1 = -2. 

The induction starts with /3 = [1, 2 . . . , fc]. We have shown 

Proposition 36. Suppose 2k < N, f3 e QN,k, P satisfies (g^, (and q"" = t'C^-'^+i) ; 
then 

Eu(p) {x) = Mu(p) {x) . 

For the specialization result, let z*^^^ be defined by zf"* = Xj if i G a (/3), and z\^'' = 
yt^pi^) if i ^ a {(3) , for variables Xi and y. 

Proposition 37. Suppose 2k < N , (3 e QN,k, P satisfies g^, (and g'" = ^-(^-'=+1) ), 
then 

k m—1 

j=i i=i 
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Proof. With the notation of Definition 

k m—1 

j=i i=i 

Replace Xi by Xi/y to finish the proof. ■ 



Example 38. Let (3 = [2,5,6,9] G fiio,4, q"'^ = 1 then 

Eu{p) [z^^^) = ix2 - yt^) (x5 - yt^) {xq - yt^) (xg - yt) 

One expects a generalization of this result for permutations of staircase partitions. 
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